In recent years, services of distributing streaming data that enables its display in real-time while acquiring data such as moving images etc. via the Internet have proliferated.
Data quality deterioration due to packet loss and delays in arrival is a problem when carrying out such a streaming distribution over the Internet. So-called error propagation occurs where the picture quality of subsequent frames is influenced if data for a certain frame is dropped due to packet loss in the case of encoding methods such as MPEG (Moving Picture Experts Group) and H.26x systems, in which differences between frames are taken. Further, the compression rate can be increased in the MPEG method using movement prediction but algorithms become complex if movement prediction is carried out and as this processing time becomes larger in proportion to the square of frame size, theoretically, a coding delay for an amount of several frames occurs. The delay time in this case is such that the time delay is on the borderline of 250 ms that is the permitted delay time when carrying out two-way communication in real-time.
On the other hand, Motion JPEG 2000 is for handling moving images by continuously reproducing the still image compression algorithm JPEG 2000 and is the successor to JPEG (Joint Photographic Expert Group). This file format is defined in part 3 of the JPEG 2000 standard standardized by the ISO (International Organization for Standardization). Motion JPEG 2000 is a moving image format that does not capture frame differences and would not experience the problems described above. Further, JPEG 2000 that is the foundation for Motion JPEG 2000 increases compression rate by using wavelength conversion and entropy encoding. As this is a moving image format that does not capture frame differences, compression and picture quality is higher than with the same Motion JPEG and DV codecs (Digital Video codec). Moreover, tolerance of various errors can be achieved with JPEG 2000 and it is considered that a moving image format such as Motion JPEG 2000 that does not take differences is appropriate for environments where packet loss occurs such as on the Internet.
In the event of carrying out streaming distribution of moving image data compressed using JPEG 2000, moving image data is distributed in packet form using RTP (Real-time Transport Protocol). “RTP Payload Format for JPEG 2000 Video streams” has been proposed as an Internet draft as an RTP format for streaming distribution employing JPEG 2000. An example of this RTP format is described with reference to FIG. 1 and FIG. 2. The RTP format shown in FIG. 1 and FIG. 2 is based on that disclosed in “draft-ietf-avt-rtp-jpg2000-00.txt”.
In FIG. 1, an RTP packet 10 is comprised of an IP header 11 containing header information for IP (Internet Protocol) use, a UDP header 12 containing header information for UDP (User Datagram Protocol) use, an RTP header 13 containing header information for usual RTP use, an RTP payload header 14 containing header information for streaming distribution of JPEG 2000 data, and JPEG 2000 data 15 distributed by streaming.
At the RTP packet 10, the IP header 11 is 20 bytes, the UDP header 12 is 8 bytes, the RTP header 13 is 12 bytes, the RTP payload header 14 is 8 bytes, and the JPEG 2000 data is in the order of 10 to 1400 bytes.
FIG. 2 is a view showing the details of the RTP header 13 and the RTP payload header 14 shown in FIG. 1.
In FIG. 2, the RTP header 13 is comprised of version bits (V) 21 of two bits, a padding bit (P) 22 of one bit, an extension bit (X) 23 of one bit, a contributing sources identifier (Contributing Sources) count bit (CC(CSRCcount)) 24 of four bits, a marker (M) 25 of one bit, a payload type (Payload type) 26 of seven bits, a sequence number (Sequence number 27) of 16 bits, an RTP time stamp (RTP timestamp) 28 of 32 bits, and a synchronization sources identifier (SSRC(Synchronization Sources)) 29 of 32 bits.
The RTP payload header 14 continuing on from the RTP header 13 is comprised of an enable bit (E) 31 of one bit, an extension bit (X) 32 of one bit, a main header (M) 33 of one bit, a tile header (T) 34 of one bit, a termination flag (L) 35 of one bit, a main header ID (Mh13 id) 36 of 3 bits, a priority (Priority) 37 of 8 bits, a tile header ID (tile_id) 38 of 16 bits, and an offset value (Fragment offset) 39 of 32 bits.
The version bits (V) 21 of the RTP packet 13 are bits showing the version of the RTP. The padding bit (P) 22 shows the presence of padding at the end of the data if this value is “1” and the extension bit 23 indicates the presence of an extension header if this value is “1”.
Further, the contributing sources identifier count bit (CC) 24 shows the number of contributing source identifiers (CSRC) contained in the RTP header 13. The marker (M) 25 is dependent on the payload type and has various uses, and the payload type (Payload type 26) expresses the type of the transmitted stream. Moreover, the sequence number (Sequence number) 27 is constructed from a number showing the packet order.
Further, the RTP time stamp (RTP timestamp) 28 indicates a clock value for this data, and the synchronization sources identifier (SSRC) 29 is a synchronization source identifier (Synchronization Sources) constituting an ID (IDentification) for uniquely identifying the sender.
The enable bit (E) 31 of the RTP payload header 14 is a JPEG 2000 payload header validation bit, and if the value of this enable bit (E) 31 is “0”, it is shown that each of the fields of the main header (M) 33, tile header (T) 34, termination flag (L) 35, main header ID (Mh_id) 36, priority (Priority) 37, and tile header ID (tile_id) 38 are not used, and these field values are all set to “0”. Further, conversely, if the value of the enable bit (E) 31 is “1”, this shows that the values of the aforementioned fields are all effective. The fields of the extension bit (X) 32 and the offset value (Fragment offset) 39 are always valid regardless of the value of the enable bit (E) 31.
The extension bit (X) indicates the continuation of the extension header if this value is “1”.
Further, the main header (M) 33 indicates the presence of a main header in a packet, the tile header (T) 34 indicates the presence of a tile header in a packet, and the termination flag (L) 35 indicates the presence of the very end of a main header or tile header in a packet.
In the event that the combination of values for the main header (M) 33, tile header (T) 34 and termination flag 35 is, in order, “101”, the data 15 of the JPEG 2000 of FIG. 1 is configured from only the main header or is configured only from the final portion within the main header which is divided into a plurality of parts. In the event that the combination of values for the main header (M) 33, tile header (T) 34 and termination flag (L) 35 is, in order, “011”, the JPEG 2000 data 15 of FIG. 1 is configured from a title header and JPEG 2000 packet (hereinafter referred to as a JP2 packet), or only a final portion within a tile header that has been divided into a plurality.
Further, in the event that the combination of values for the main header (M) 33, tile header (T) 34 and termination flag (L) 35 is, in order, “100”, the JPEG 2000 data of FIG. 1 is configured from the first portion within a main header divided into a plurality or only a middle portion. In the event that the combination of values for the main header (M) 33, tile header (T) 34 and termination flag (L) 35 is, in order, “010”, the JPEG 2000 data 15 of FIG. 1 is configured from a first portion within the tile headers divided into a plurality or a middle portion only.
Moreover, in the event that the combination of values for the main header (M) 33, tile header (T) 34 and termination flag (L) 35 is, in order, “111”, the JPEG data 15 of FIG. 1 is configured from a main header and a tile header, or is configured from a main header, tile header and JP2 packet.
Namely, through combinations of these values as shown above, the main header (M) 33, tile header (T) 34 and termination flag (L) 35 indicate what kind of header section is configured at a packet.
An ID of the main domain is shown at the main header ID (Mh_id) 36 of FIG. 2, and contains values (“1” to “7”) that are incremented every time the main header changes. The value “0” is set if this is not-yet used.
Priority (Priority) 37 expresses the level of importance (1 to 254) of the JP2 packet within the RTP packet.
The tile header ID (tile_id) 38 is set with a tile header within the payload and the title ID (0x0000 to 0xffff) to which the JP2 packet belongs. In the event that only a main header exists, the value “0” can be set to the tile header ID38.
The offset value (Fragment offset) 39 indicates an offset value within the JPEG2000 frame data, i.e. an octet number from the header of the frame.
When the priority value set at the priority (Priority) 37 of the RTP payload header 14 shown in FIG. 2 is “1”, the priority is the highest, with the priority falling every time the value is increased, so that the priority in the case of “254” gives the lowest priority.
In, for example, the event that a user views only part of an entire image of an image with a broad range such as an omni-directional image, the priority value is a value referenced while decoding only image data corresponding to a requested portion (containing a portion for this range as requested) rather than image data corresponding to the whole image. Further, it is there possible to carry out partial distribution by distributing only data corresponding to part of the image by having the distribution server carrying out the streaming distribution refer to this priority value.
Moreover, for example, in the event of terminals connected to networks of narrow bandwidths or carrying out streaming distribution with respect to terminals of low playback performance, only the first JPEG 2000 data within a single image frame encoded using layer encoding is decoded. This means that it is possible to output images of low resolution and images of coarse picture quality and overflow can therefore be prevented.
The distribution server refers to the priority value every JP2 packet, and is capable of carrying out scaleable distribution the distributes only part of the data.
Further, this priority value can be set based on a priority mapping table prepared each session. When the JPEG 2000 data (JP2 packet) in the packet is a header section, the priority value is set with the value “0” for the highest priority, and in the event of a data section, values from “1” to “254” can be set. In the event that a priority mapping table does not exist, a value “255” indicating that this field does not yet exist can be set.
FIG. 3 is a view showing an example of a priority table.
In FIG. 3, a priority table 41 is a correspondence table for a layer (L (layer)), resolution (R(resolution)), component (C (component)) and precinct (P (precinct)), and priority value (Priority). The priority table 41 is a table for the case where the priority value is set taking a layer as a reference.
FIG. 4 is a view showing a further example of a priority table.
The priority table 51 shown in FIG. 4 is a table for the case where the priority value is set taking resolution as a reference.
However, in the event that, for example, a single JPEG 2000 frame is configured from a large number of JP2 packets and the priority values are set to be fine, it is necessary to set priority for all of the JP2 packets. Because of this, it is necessary to be aware of the number of JP2 packets and it is necessary to set priority values just for this number of packets, which requires a complex operation.
FIG. 5 is a view showing an example of a priority table occurring in this case.
In the event that the priority value is set to fine, as with the priority table 61 shown in FIG. 5, the amount of data for the priority table prepared in advance is substantial, and the amount of work involved in generating this table increases.